Skip to content

Conversation

@travissluka
Copy link
Contributor

@travissluka travissluka commented Oct 3, 2023

Description

changes:

  • the ctest test_soca_3dvar_godas now uses VertInterp instead of InsituTemperature and so should make use of VADER for the TL/AD
  • added a bunch of hacky looking code to do some variable renaming to make VADER happy. Don't look at this too much yet, it will be cleaned up a bit before this PR is merged, and will go away entirely when a follow up PR is done to clean up the variable names.

@Dooruk you should be able to use this to test your vader branch. In order for it to compile, I added a changeVarAD() empty stub, and commented out all the code in changeVarTL() and changeVarTraj() because they currently contain bugs.

Dependencies

build-group=https://github.com/JCSDA-internal/vader/pull/208

@Dooruk
Copy link
Collaborator

Dooruk commented Oct 4, 2023

Thanks @travissluka !

It's still not clear to me what atlas grid type would handle MOM6 tripolar grid (which structured type). For instance, ORCA has an add-on for atlas:

https://github.com/ecmwf/atlas-orca

Does tripolar MOM6 grid differ from ORCA dramatically that we don't need such add-on?

@danholdaway suggested using an atmospheric gaussian grid that VADER tests currently work for and just replace existing values with ocean variables. I can do that but i would like to have a permanent solution eventually.

@travissluka
Copy link
Contributor Author

@Dooruk I agree with Dan that it's probably easier to just use an existing gaussian grid. I don't think we need to go through the trouble of making sure the same tri-polar grid works with the vader ctests

@eap
Copy link
Contributor

eap commented Aug 31, 2025

Tests on this PR are disabled until you merge or run git cherry-pick 91300a4

@shlyaeva shlyaeva marked this pull request as ready for review December 30, 2025 17:39
@shlyaeva
Copy link
Collaborator

The code here is now updated to use vader recipe that @Dooruk introduced in https://github.com/JCSDA-internal/vader/pull/208 in linear variable changes.

There are still a few hacks here and there that I borrowed from @travissluka's original code, that we may be able to remove / simplify in the future. Another important point to note is that this approach (using VertInterp with vader variable change instead of InsituTemperature) doesn't work well with dual resolution. This is due to missing points generated by the interpolation to a different resolution that we don't handle properly (in any variable change to my knowledge). A disturbing problem for another day.

I think we should test all our current applications with this branch properly, before we proceed with merging, but please review when you have a minute. I'll start testing with our applications once I have some feedback.

@Dooruk
Copy link
Collaborator

Dooruk commented Dec 30, 2025

@shlyaeva thank you so much for moving this forward this is a lot of work in multiple repos. I like the idea of separating TL/AD from the NL part. I follow the changes in LinearVar.cc but don't have much to suggest there.

I tested the SOCA + VADER add_gswocean_tl branches using my sandbox setup against a build from 2 weeks ago. I used 0.25-deg resolution, Static 3DVAR, 10 iterations, XBT temperature and ARGO composite operator and the norm reduction results are identical. With VADER runtime was slightly longer. I suspect with more observation types this should favor towards VADER? I could test with a more realistic setup as well when I get a chance to give an approval.

@shlyaeva
Copy link
Collaborator

shlyaeva commented Jan 7, 2026

thank you so much for testing @Dooruk! can you share the logs from the two runs? I didn't expect the runtime to be much different, curious to see where it's happening. I'll test in our setup too.

@Dooruk
Copy link
Collaborator

Dooruk commented Jan 12, 2026

thank you so much for testing @Dooruk! can you share the logs from the two runs? I didn't expect the runtime to be much different, curious to see where it's happening. I'll test in our setup too.

Sure, if you want the full logs I can email them but here are the OOPS bits:

with vader:

Run: Finishing oops::Variational<SOCA, UFO and IODA observations>
OOPS_STATS
OOPS_STATS ----------------------------------------------------------------------------------
OOPS_STATS --------------------------- Object counts ----------------------------------------
OOPS_STATS ----------------------------------------------------------------------------------
OOPS_STATS                                      Total  Simult. Remain    Avg (Mb)    HWM (Mb)
OOPS_STATS ioda::ObsDataVector             :       33      13
OOPS_STATS ioda::ObsVector                 :      215     111
OOPS_STATS oops::ControlIncrement          :       43       8                4.33       34.67
OOPS_STATS oops::ControlVariable           :        4       3
OOPS_STATS oops::Diffusion                 :        3       2
OOPS_STATS oops::GeoVaLs                   :       81       1
OOPS_STATS oops::Geometry                  :        2       2
OOPS_STATS oops::GetValues                 :        9       6
OOPS_STATS oops::Increment                 :       91       9                4.14       39.00
OOPS_STATS oops::LinearObsOperator         :        3       3
OOPS_STATS oops::LocalInterpolator         :     6480    4320
OOPS_STATS oops::ModelAuxControl           :        4       3
OOPS_STATS oops::ModelAuxCovariance        :        1       1
OOPS_STATS oops::ModelAuxIncrement         :       43       8
OOPS_STATS oops::ModelData                 :        1       1
OOPS_STATS oops::ObsAuxControl             :       12       9
OOPS_STATS oops::ObsAuxCovariance          :        3       3
OOPS_STATS oops::ObsAuxIncrement           :      129      24
OOPS_STATS oops::ObsAuxIncrements          :       43       8
OOPS_STATS oops::ObsDataVector             :       18      12
OOPS_STATS oops::ObsDiagnostics            :        6       1
OOPS_STATS oops::ObsError                  :        3       3
OOPS_STATS oops::ObsOperator               :        6       6
OOPS_STATS oops::ObsSpace                  :        3       3
OOPS_STATS oops::ObsSpaces                 :        1       1
OOPS_STATS oops::ObsVector                 :      215     108                0.00        0.01
OOPS_STATS oops::SampledLocations          :       24       8
OOPS_STATS oops::State                     :       10       5                5.79       28.96
OOPS_STATS oops::UnstructuredInterpolator  :     6480    4320
OOPS_STATS oops::VariableChange            :        3       1
OOPS_STATS saber::ErrorCovariance          :        1       1
OOPS_STATS soca::Geometry                  :        2       2
OOPS_STATS soca::Increment                 :      163      10
OOPS_STATS soca::ModelBias                 :        4       3
OOPS_STATS soca::ModelBiasCovariance       :        1       1
OOPS_STATS soca::State                     :       18       7
OOPS_STATS ufo::BackgroundCheck            :        1       1
OOPS_STATS ufo::FinalCheck                 :        6       4
OOPS_STATS ufo::GeoVaLs                    :       90       2
OOPS_STATS ufo::ObsADT                     :        2       2
OOPS_STATS ufo::ObsADTTLAD                 :        1       1
OOPS_STATS ufo::ObsBias                    :       12       9
OOPS_STATS ufo::ObsBiasCovariance          :        3       3
OOPS_STATS ufo::ObsBoundsCheck             :        3       3
OOPS_STATS ufo::ObsComposite               :        2       2
OOPS_STATS ufo::ObsCompositeTLAD           :        1       1
OOPS_STATS ufo::ObsDomainCheck             :        8       8
OOPS_STATS ufo::ObsFilter                  :       44      21
OOPS_STATS ufo::ObsFilterData              :       18      16
OOPS_STATS ufo::ObsInsituTemperature       :        4       4
OOPS_STATS ufo::ObsInsituTemperatureTLAD   :        2       2
OOPS_STATS ufo::ObsVertInterp              :        2       2
OOPS_STATS ufo::ObsVertInterpTLAD          :        1       1
OOPS_STATS ufo::SampledLocations           :       12       7
OOPS_STATS ----------------------------- Object counts --------------------------------------
OOPS_STATS
OOPS_STATS --------------------------------------------------------------------------------------------
OOPS_STATS ------------------------------------ Timing Statistics -------------------------------------
OOPS_STATS --------------------------------------------------------------------------------------------
OOPS_STATS Name                                                :   total (ms)   count    time/call (ms)
OOPS_STATS ObsFilter::Background Check::ObsFilter              :         0.06       1            0.0604
OOPS_STATS ObsFilter::Background Check::postFilter             :         0.92       1            0.9166
OOPS_STATS ObsFilter::Background Check::preProcess             :         0.00       1            0.0006
OOPS_STATS ObsFilter::Background Check::priorFilter            :         0.00       1            0.0006
OOPS_STATS ObsFilter::Bounds Check::ObsFilter                  :         0.22       3            0.0744
OOPS_STATS ObsFilter::Bounds Check::postFilter                 :         0.00       3            0.0002
OOPS_STATS ObsFilter::Bounds Check::preProcess                 :         0.31       3            0.1019
OOPS_STATS ObsFilter::Bounds Check::priorFilter                :         0.00       3            0.0007
OOPS_STATS ObsFilter::Domain Check::ObsFilter                  :         0.72       8            0.0898
OOPS_STATS ObsFilter::Domain Check::postFilter                 :         0.00       8            0.0003
OOPS_STATS ObsFilter::Domain Check::preProcess                 :         0.65       8            0.0811
OOPS_STATS ObsFilter::Domain Check::priorFilter                :         0.37       8            0.0464
OOPS_STATS ObsFilter::Final Check::ObsFilter                   :         0.02       6            0.0040
OOPS_STATS ObsFilter::Final Check::postFilter                  :         0.03       6            0.0056
OOPS_STATS ObsFilter::Final Check::preProcess                  :         0.00       6            0.0005
OOPS_STATS ObsFilter::Final Check::priorFilter                 :         0.01       6            0.0012
OOPS_STATS ObsFilter::QCmanager::ObsFilter                     :         0.23       6            0.0383
OOPS_STATS ObsFilter::QCmanager::postFilter                    :         0.01       6            0.0020
OOPS_STATS ObsFilter::QCmanager::preProcess                    :         0.00       6            0.0002
OOPS_STATS ObsFilter::QCmanager::priorFilter                   :         0.00       6            0.0002
OOPS_STATS Vader::Vader                                        :         0.28       5            0.0560
OOPS_STATS Vader::changeVar                                    :         4.74       3            1.5810
OOPS_STATS Vader::changeVarTraj                                :        11.43       2            5.7169
OOPS_STATS oops::Covariance::SABER::Constructor                :        46.15       1           46.1541
OOPS_STATS oops::Covariance::SABER::multiply                   :    116215.71      12         9684.6425
OOPS_STATS oops::Diffusion::Diffusion                          :         2.51       3            0.8362
OOPS_STATS oops::Diffusion::calculateDerivedGeom               :         2.80       2            1.3988
OOPS_STATS oops::Diffusion::multiply                           :       916.56       2          458.2818
OOPS_STATS oops::Diffusion::multiplySqrtAD                     :     61629.39      36         1711.9274
OOPS_STATS oops::Diffusion::multiplySqrtTL                     :     53090.38      36         1474.7327
OOPS_STATS oops::Diffusion::setParameters                      :       161.78       3           53.9283
OOPS_STATS oops::GeoVaLs::GeoVaLs                              :         1.48      81            0.0182
OOPS_STATS oops::GeoVaLs::fill                                 :         0.42     676            0.0006
OOPS_STATS oops::GeoVaLs::fillAD                               :         0.30     468            0.0006
OOPS_STATS oops::GeoVaLs::~GeoVaLs                             :         0.34      81            0.0042
OOPS_STATS oops::Geometry::Geometry                            :      5356.87       2         2678.4347
OOPS_STATS oops::Geometry::variableSizes                       :         0.79      31            0.0255
OOPS_STATS oops::Geometry::~Geometry                           :         0.35       2            0.1772
OOPS_STATS oops::GeometryData::setGlobalTree                   :      2448.36       2         1224.1823
OOPS_STATS oops::GeometryData::setLocalTree                    :         2.54       2            1.2695
OOPS_STATS oops::GeometryData::setMeshAndTriangulation         :         1.05       2            0.5274
OOPS_STATS oops::GetValues::GetValues                          :      6566.33       9          729.5923
OOPS_STATS oops::GetValues::fillGeoVaLs                        :        64.42       9            7.1574
OOPS_STATS oops::GetValues::fillGeoVaLsAD                      :         2.95      36            0.0818
OOPS_STATS oops::GetValues::fillGeoVaLsTL                      :        84.67      36            2.3519
OOPS_STATS oops::GetValues::finalize                           :         0.20       9            0.0223
OOPS_STATS oops::GetValues::finalizeAD                         :        22.03      36            0.6119
OOPS_STATS oops::GetValues::finalizeTL                         :         0.62      36            0.0171
OOPS_STATS oops::GetValues::initialize                         :         0.20       9            0.0218
OOPS_STATS oops::GetValues::initializeAD                       :         0.38      36            0.0106
OOPS_STATS oops::GetValues::initializeTL                       :         0.86      36            0.0239
OOPS_STATS oops::GetValues::process                            :        94.07       9           10.4519
OOPS_STATS oops::GetValues::processAD                          :       190.97      36            5.3047
OOPS_STATS oops::GetValues::processTL                          :       232.81      36            6.4671
OOPS_STATS oops::Increment::Increment                          :       449.01      91            4.9341
OOPS_STATS oops::Increment::axpy                               :         4.83       1            4.8338
OOPS_STATS oops::Increment::diff                               :        18.35       3            6.1161
OOPS_STATS oops::Increment::dot_product_with                   :        47.21       3           15.7382
OOPS_STATS oops::Increment::fromFieldSet                       :        74.95      24            3.1230
OOPS_STATS oops::Increment::operator+=                         :        61.81      14            4.4148
OOPS_STATS oops::Increment::operator+=(State, Increment)       :         4.52       1            4.5170
OOPS_STATS oops::Increment::operator=                          :       214.51      50            4.2902
OOPS_STATS oops::Increment::print                              :        39.03       5            7.8062
OOPS_STATS oops::Increment::serialSize                         :         0.02      43            0.0005
OOPS_STATS oops::Increment::toFieldSet                         :         0.54      36            0.0151
OOPS_STATS oops::Increment::write                              :      4994.19       1         4994.1945
OOPS_STATS oops::Increment::zero                               :        12.66      24            0.5275
OOPS_STATS oops::Increment::~Increment                         :         3.97      91            0.0436
OOPS_STATS oops::LinearObsOper::ADT::LinearObsOperator         :         0.05       1            0.0480
OOPS_STATS oops::LinearObsOper::ADT::requiredVars              :         0.04      26            0.0014
OOPS_STATS oops::LinearObsOper::ADT::setTrajectory             :         0.53       1            0.5254
OOPS_STATS oops::LinearObsOper::ADT::simulateObsAD             :        11.68      12            0.9737
OOPS_STATS oops::LinearObsOper::ADT::simulateObsTL             :       182.25      12           15.1871
OOPS_STATS oops::LinearObsOper::ADT::~LinearObsOperator        :         0.01       1            0.0077
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::LinearObs :         0.52       1            0.5210
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::requiredV :         0.02      26            0.0007
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::setTrajec :         0.48       1            0.4832
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::simulateO :         1.09      12            0.0909
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::simulateO :         1.09      12            0.0905
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::~LinearOb :         0.04       1            0.0370
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::LinearObsOpe :         0.04       1            0.0367
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::requiredVars :         0.02      26            0.0008
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::setTrajector :         0.39       1            0.3884
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::simulateObsA :         0.18      12            0.0149
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::simulateObsT :         0.14      12            0.0121
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::~LinearObsOp :         0.01       1            0.0131
OOPS_STATS oops::LinearVariableChange::LinearVariableChange    :         0.43       2            0.2173
OOPS_STATS oops::LinearVariableChange::changeVarAD             :       477.11      24           19.8795
OOPS_STATS oops::LinearVariableChange::changeVarTL             :       317.12      24           13.2133
OOPS_STATS oops::LinearVariableChange::changeVarTraj           :        73.23       2           36.6130
OOPS_STATS oops::LinearVariableChange::~LinearVariableChange   :         0.48       2            0.2412
OOPS_STATS oops::LocalInterpolator::LocalInterpolator          :         8.32    6480            0.0013
OOPS_STATS oops::LocalInterpolator::apply                      :        87.62    6480            0.0135
OOPS_STATS oops::LocalInterpolator::applyAD                    :       164.57   25920            0.0063
OOPS_STATS oops::LocalInterpolator::applyTL                    :       206.57   25920            0.0080
OOPS_STATS oops::LocalInterpolator::preprocess                 :        95.05      15            6.3368
OOPS_STATS oops::LocalInterpolator::preprocessAD               :        55.23      12            4.6026
OOPS_STATS oops::LocalInterpolator::~LocalInterpolator         :         2.01    6480            0.0003
OOPS_STATS oops::ModelAuxControl::ModelAuxControl              :         0.01       4            0.0016
OOPS_STATS oops::ModelAuxControl::print                        :         0.00       3            0.0007
OOPS_STATS oops::ModelAuxControl::~ModelAuxControl             :         0.00       4            0.0008
OOPS_STATS oops::ModelAuxCovariance::ModelAuxCovariance        :         0.00       1            0.0023
OOPS_STATS oops::ModelAuxCovariance::linearize                 :         0.00       1            0.0004
OOPS_STATS oops::ModelAuxCovariance::multiply                  :         0.00      12            0.0002
OOPS_STATS oops::ModelAuxCovariance::~ModelAuxCovariance       :         0.00       1            0.0011
OOPS_STATS oops::ModelAuxIncrement::ModelAuxIncrement          :         0.05      43            0.0013
OOPS_STATS oops::ModelAuxIncrement::axpy                       :         0.00       1            0.0002
OOPS_STATS oops::ModelAuxIncrement::diff                       :         0.00       3            0.0002
OOPS_STATS oops::ModelAuxIncrement::dot_product_with           :         0.00       3            0.0003
OOPS_STATS oops::ModelAuxIncrement::operator+=                 :         0.00       2            0.0002
OOPS_STATS oops::ModelAuxIncrement::operator+=ModelAuxControl  :         0.00       1            0.0003
OOPS_STATS oops::ModelAuxIncrement::print                      :         0.00       5            0.0010
OOPS_STATS oops::ModelAuxIncrement::serialSize                 :         0.01      43            0.0001
OOPS_STATS oops::ModelAuxIncrement::write                      :         0.00       1            0.0003
OOPS_STATS oops::ModelAuxIncrement::zero                       :         0.00      12            0.0002
OOPS_STATS oops::ModelAuxIncrement::~ModelAuxIncrement         :         0.02      43            0.0004
OOPS_STATS oops::ModelData::ModelData                          :         0.00       1            0.0005
OOPS_STATS oops::ModelData::modelData                          :         0.00       1            0.0004
OOPS_STATS oops::ModelData::~ModelData                         :         0.00       1            0.0011
OOPS_STATS oops::ObsAuxControl::ObsAuxControl                  :         1.13      12            0.0944
OOPS_STATS oops::ObsAuxControl::print                          :         0.16       9            0.0183
OOPS_STATS oops::ObsAuxControl::requiredHdiagnostics           :         0.00       6            0.0002
OOPS_STATS oops::ObsAuxControl::requiredVars                   :         0.01      18            0.0004
OOPS_STATS oops::ObsAuxControl::write                          :         0.06       3            0.0215
OOPS_STATS oops::ObsAuxControl::~ObsAuxControl                 :         0.03      12            0.0022
OOPS_STATS oops::ObsAuxCovariance::ObsAuxCovariance            :         0.16       3            0.0523
OOPS_STATS oops::ObsAuxCovariance::linearize                   :         0.03       3            0.0092
OOPS_STATS oops::ObsAuxCovariance::multiply                    :         0.11      36            0.0031
OOPS_STATS oops::ObsAuxCovariance::write                       :         0.10       3            0.0319
OOPS_STATS oops::ObsAuxCovariance::~ObsAuxCovariance           :         0.01       3            0.0026
OOPS_STATS oops::ObsAuxIncrement::ObsAuxIncrement              :         2.48     129            0.0192
OOPS_STATS oops::ObsAuxIncrement::axpy                         :         0.01       3            0.0049
OOPS_STATS oops::ObsAuxIncrement::diff                         :         0.05       9            0.0052
OOPS_STATS oops::ObsAuxIncrement::dot_product_with             :         0.01       9            0.0010
OOPS_STATS oops::ObsAuxIncrement::operator+=                   :         0.03       6            0.0048
OOPS_STATS oops::ObsAuxIncrement::print                        :         0.01      15            0.0004
OOPS_STATS oops::ObsAuxIncrement::serialSize                   :         0.02     258            0.0001
OOPS_STATS oops::ObsAuxIncrement::zero                         :         0.02      36            0.0005
OOPS_STATS oops::ObsAuxIncrement::~ObsAuxIncrement             :         0.07     129            0.0005
OOPS_STATS oops::ObsDataVector::ObsDataVector                  :         0.12      18            0.0065
OOPS_STATS oops::ObsDataVector::mask                           :         0.01       6            0.0009
OOPS_STATS oops::ObsDataVector::operator=                      :         0.02       9            0.0018
OOPS_STATS oops::ObsDataVector::print                          :         3.22       6            0.5360
OOPS_STATS oops::ObsDataVector::save                           :         0.44      12            0.0367
OOPS_STATS oops::ObsDataVector::~ObsDataVector                 :         0.02      18            0.0012
OOPS_STATS oops::ObsDiagnostics::ObsDiagnostics                :         0.07       6            0.0124
OOPS_STATS oops::ObsDiagnostics::~ObsDiagnostics               :         0.04       6            0.0059
OOPS_STATS oops::ObsError::getRMSE                             :         0.51       8            0.0633
OOPS_STATS oops::ObsError::inverseMultiply                     :         0.03      36            0.0009
OOPS_STATS oops::ObsError::obserrors                           :         0.39      14            0.0281
OOPS_STATS oops::ObsError::update                              :         6.62       6            1.1036
OOPS_STATS oops::ObsError::~ObsError                           :         0.02       3            0.0070
OOPS_STATS oops::ObsOperator::ADT::ObsOperator                 :         3.82       2            1.9088
OOPS_STATS oops::ObsOperator::ADT::computeReducedVars          :         0.12       3            0.0390
OOPS_STATS oops::ObsOperator::ADT::locations                   :         0.65       3            0.2153
OOPS_STATS oops::ObsOperator::ADT::requiredVars                :         0.00       3            0.0009
OOPS_STATS oops::ObsOperator::ADT::simulateObs                 :        46.26       2           23.1288
OOPS_STATS oops::ObsOperator::ADT::~ObsOperator                :         0.07       2            0.0334
OOPS_STATS oops::ObsOperator::insitu_profile_argo::ObsOperator :         5.46       2            2.7302
OOPS_STATS oops::ObsOperator::insitu_profile_argo::computeRedu :         0.04       3            0.0124
OOPS_STATS oops::ObsOperator::insitu_profile_argo::locations   :         0.44       3            0.1459
OOPS_STATS oops::ObsOperator::insitu_profile_argo::requiredVar :         0.00       3            0.0008
OOPS_STATS oops::ObsOperator::insitu_profile_argo::simulateObs :         4.51       2            2.2529
OOPS_STATS oops::ObsOperator::insitu_profile_argo::~ObsOperato :         0.04       2            0.0178
OOPS_STATS oops::ObsOperator::temp_profile_xbt::ObsOperator    :         0.66       2            0.3292
OOPS_STATS oops::ObsOperator::temp_profile_xbt::computeReduced :         0.04       3            0.0138
OOPS_STATS oops::ObsOperator::temp_profile_xbt::locations      :         0.34       3            0.1118
OOPS_STATS oops::ObsOperator::temp_profile_xbt::requiredVars   :         0.00       3            0.0007
OOPS_STATS oops::ObsOperator::temp_profile_xbt::simulateObs    :         0.13       2            0.0665
OOPS_STATS oops::ObsOperator::temp_profile_xbt::~ObsOperator   :         0.02       2            0.0081
OOPS_STATS oops::ObsSpace::ObsSpace                            :       736.91       3          245.6371
OOPS_STATS oops::ObsSpace::obsvariables                        :         0.00      12            0.0001
OOPS_STATS oops::ObsSpace::save                                :       945.70       3          315.2342
OOPS_STATS oops::ObsSpace::~ObsSpace                           :         0.25       3            0.0826
OOPS_STATS oops::ObsVector::ObsVector                          :         0.40     209            0.0019
OOPS_STATS oops::ObsVector::axpy                               :         0.09     228            0.0004
OOPS_STATS oops::ObsVector::dot_product                        :        35.31     242            0.1459
OOPS_STATS oops::ObsVector::info                               :         1.10      18            0.0611
OOPS_STATS oops::ObsVector::operator*=                         :         0.05     132            0.0003
OOPS_STATS oops::ObsVector::operator+=                         :         0.02      90            0.0003
OOPS_STATS oops::ObsVector::operator-=                         :         0.00       6            0.0006
OOPS_STATS oops::ObsVector::operator=                          :         0.09     144            0.0006
OOPS_STATS oops::ObsVector::rms                                :         0.23       8            0.0292
OOPS_STATS oops::ObsVector::save                               :         0.46      18            0.0255
OOPS_STATS oops::ObsVector::zero                               :         0.01      45            0.0003
OOPS_STATS oops::ObsVector::~ObsVector                         :         0.17     209            0.0008
OOPS_STATS oops::Parameters::deserialize                       :         8.43     173            0.0487
OOPS_STATS oops::Parameters::serialize                         :         1.88      67            0.0280
OOPS_STATS oops::SampledLocations::SampledLocations            :         0.00      12            0.0003
OOPS_STATS oops::SampledLocations::latitudes                   :         0.00       9            0.0001
OOPS_STATS oops::SampledLocations::longitudes                  :         0.00       9            0.0001
OOPS_STATS oops::SampledLocations::times                       :         0.00       9            0.0001
OOPS_STATS oops::SampledLocations::~SampledLocations           :         0.31      24            0.0128
OOPS_STATS oops::State::State                                  :     16158.13      10         1615.8126
OOPS_STATS oops::State::print                                  :        21.43       3            7.1446
OOPS_STATS oops::State::toFieldSet                             :         0.16       5            0.0312
OOPS_STATS oops::State::write                                  :      4172.00       1         4171.9956
OOPS_STATS oops::State::~State                                 :         0.51      10            0.0514
OOPS_STATS oops::UnstructuredInterpolator::UnstructuredInterpo :         3.67    6480            0.0006
OOPS_STATS oops::UnstructuredInterpolator::apply               :       275.67   32400            0.0085
OOPS_STATS oops::UnstructuredInterpolator::applyAD             :       149.31   25920            0.0058
OOPS_STATS oops::VariableChange::VariableChange                :         0.61       3            0.2037
OOPS_STATS oops::VariableChange::changeVar                     :        59.05       3           19.6827
OOPS_STATS oops::VariableChange::~VariableChange               :         0.03       3            0.0096
OOPS_STATS oops::util::FieldSetHelpers::readFieldSet           :      5937.49       3         1979.1640
OOPS_STATS saber::ErrorCovariance::ErrorCovariance             :      8729.37       1         8729.3663
OOPS_STATS saber::ErrorCovariance::doMultiply                  :    115439.88      12         9619.9903
OOPS_STATS saber::ErrorCovariance::~ErrorCovariance            :         0.00       1            0.0002
OOPS_STATS soca::Balance::Balance                              :        14.16       1           14.1643
OOPS_STATS soca::Balance::multiply                             :       116.62      12            9.7183
OOPS_STATS soca::Balance::multiplyAD                           :        95.66      12            7.9719
OOPS_STATS soca::BkgErrFilt::BkgErrFilt                        :         0.23       1            0.2263
OOPS_STATS soca::BkgErrFilt::multiply                          :        62.03      24            2.5844
OOPS_STATS soca::FieldsMetadata::FieldsMetadata                :        36.25       2           18.1232
OOPS_STATS soca::LinearModel2GeoVaLs::multiply                 :        18.85      12            1.5705
OOPS_STATS soca::LinearModel2GeoVaLs::multiplyAD               :        25.43      12            2.1192
OOPS_STATS soca::Model2GeoVaLs::Model2GeoVaLs                  :         0.00       3            0.0003
OOPS_STATS soca::Model2GeoVaLs::changeVar                      :         7.95       3            2.6485
OOPS_STATS soca::ParametricOceanStdDev::ParametricOceanStdDev  :      2750.85       1         2750.8511
OOPS_STATS soca::ParametricOceanStdDev::multiply               :        32.99      12            2.7494
OOPS_STATS soca::ParametricOceanStdDev::multiplyAD             :        28.91      12            2.4089
OOPS_STATS soca::VariableChange::VariableChange                :         0.58       3            0.1921
OOPS_STATS soca::VariableChange::changeVar                     :        59.00       3           19.6652
OOPS_STATS soca::readNcAndInterp::readNcAndInterp              :      1659.78       1         1659.7808
OOPS_STATS ufo::ObsFilter::ObsFilter                           :         1.29      24            0.0538
OOPS_STATS ufo::ObsFilter::checkFilterData                     :         0.02      72            0.0003
OOPS_STATS ufo::ObsFilter::postFilter                          :         0.99      24            0.0411
OOPS_STATS ufo::ObsFilter::preProcess                          :         0.98      24            0.0409
OOPS_STATS ufo::ObsFilter::priorFilter                         :         0.41      24            0.0170
OOPS_STATS ufo::ObsFilter::~ObsFilter                          :         6.35      44            0.1444
OOPS_STATS util::Timers::Total                                 :    168779.50       1       168779.5001
OOPS_STATS util::Timers::measured                              :    168489.11       1       168489.1113
OOPS_STATS ------------------------------------ Timing Statistics -------------------------------------

OOPS_STATS ------------------------------------------------------------------------------------------------------------------
OOPS_STATS ---------------------------------- Parallel Timing Statistics ( 720 MPI tasks) -----------------------------------
OOPS_STATS ------------------------------------------------------------------------------------------------------------------
OOPS_STATS Name                                                :     min (ms)    max (ms)    avg (ms)     % total   imbal (%)
OOPS_STATS oops::Covariance::SABER::multiply                   :    115851.59   116303.17   116208.83       68.85        0.39
OOPS_STATS oops::Diffusion::multiply                           :       876.98      998.50      903.41        0.54       13.45
OOPS_STATS oops::Diffusion::multiplySqrtAD                     :     61189.72    61691.71    61426.23       36.39        0.82
OOPS_STATS oops::Diffusion::multiplySqrtTL                     :     53031.10    53502.78    53268.84       31.56        0.89
OOPS_STATS oops::Geometry::Geometry                            :      4997.38     5392.07     5122.60        3.04        7.70
OOPS_STATS oops::GeometryData::setGlobalTree                   :      2067.23     2512.96     2422.04        1.44       18.40
OOPS_STATS oops::GetValues::GetValues                          :       101.03    22622.33    11611.28        6.88      193.96
OOPS_STATS oops::GetValues::processAD                          :       187.79      207.47      192.77        0.11       10.21
OOPS_STATS oops::GetValues::processTL                          :       227.82      250.84      234.47        0.14        9.82
OOPS_STATS oops::Increment::Increment                          :       107.07      510.28      447.37        0.27       90.13
OOPS_STATS oops::Increment::operator=                          :       120.65      245.79      223.13        0.13       56.09
OOPS_STATS oops::Increment::write                              :       525.82     4994.19     2728.94        1.62      163.74
OOPS_STATS oops::LinearVariableChange::changeVarAD             :       204.89      552.24      456.94        0.27       76.02
OOPS_STATS oops::LinearVariableChange::changeVarTL             :       134.77      353.67      317.23        0.19       69.00
OOPS_STATS oops::LinearVariableChange::changeVarTraj           :        69.76      299.18      170.53        0.10      134.54
OOPS_STATS oops::LocalInterpolator::applyTL                    :       203.06      225.92      209.06        0.12       10.93
OOPS_STATS oops::LocalInterpolator::preprocess                 :        50.72     3601.52     1715.52        1.02      206.98
OOPS_STATS oops::ObsOperator::ADT::simulateObs                 :        19.30      560.27      518.30        0.31      104.38
OOPS_STATS oops::ObsSpace::ObsSpace                            :       698.11      929.61      749.00        0.44       30.91
OOPS_STATS oops::ObsSpace::save                                :       833.25     1055.57      947.39        0.56       23.47
OOPS_STATS oops::State::State                                  :       260.35    22617.24    11138.30        6.60      200.72
OOPS_STATS oops::State::print                                  :        21.43     4495.34     2290.80        1.36      195.30
OOPS_STATS oops::State::write                                  :       464.00     4172.00     2027.03        1.20      182.93
OOPS_STATS oops::UnstructuredInterpolator::apply               :       271.11      301.00      278.46        0.16       10.74
OOPS_STATS oops::util::FieldSetHelpers::readFieldSet           :      5885.15     5956.43     5941.58        3.52        1.20
OOPS_STATS saber::ErrorCovariance::ErrorCovariance             :      8559.94     8736.15     8663.33        5.13        2.03
OOPS_STATS saber::ErrorCovariance::doMultiply                  :    115217.64   115779.83   115444.78       68.40        0.49
OOPS_STATS util::Timers::Total                                 :    168770.71   168781.22   168778.62      100.00        0.01
OOPS_STATS util::Timers::measured                              :    168102.76   168574.31   168291.75       99.71        0.28
OOPS_STATS ---------------------------------- Parallel Timing Statistics ( 720 MPI tasks) -----------------------------------

OOPS_STATS Run end                                  - Runtime:    174.99 sec,  Memory: total:   681.77 Gb, per task: min =   882.70 Mb, max =  4311.54 Mb
Run: Finishing oops::Variational<SOCA, UFO and IODA observations> with status = 0
OOPS Ending   2025-12-30 14:45:06 (UTC-0500)

Prior to vader:

Run: Finishing oops::Variational<SOCA, UFO and IODA observations>
OOPS_STATS
OOPS_STATS ----------------------------------------------------------------------------------
OOPS_STATS --------------------------- Object counts ----------------------------------------
OOPS_STATS ----------------------------------------------------------------------------------
OOPS_STATS                                      Total  Simult. Remain    Avg (Mb)    HWM (Mb)
OOPS_STATS ioda::ObsDataVector             :       33      13
OOPS_STATS ioda::ObsVector                 :      215     111
OOPS_STATS oops::ControlIncrement          :       43       8                4.33       34.67
OOPS_STATS oops::ControlVariable           :        4       3
OOPS_STATS oops::Diffusion                 :        3       2
OOPS_STATS oops::GeoVaLs                   :       81       1
OOPS_STATS oops::Geometry                  :        2       2
OOPS_STATS oops::GetValues                 :        9       6
OOPS_STATS oops::Increment                 :       91       9                4.14       39.00
OOPS_STATS oops::LinearObsOperator         :        3       3
OOPS_STATS oops::LocalInterpolator         :     6480    4320
OOPS_STATS oops::ModelAuxControl           :        4       3
OOPS_STATS oops::ModelAuxCovariance        :        1       1
OOPS_STATS oops::ModelAuxIncrement         :       43       8
OOPS_STATS oops::ModelData                 :        1       1
OOPS_STATS oops::ObsAuxControl             :       12       9
OOPS_STATS oops::ObsAuxCovariance          :        3       3
OOPS_STATS oops::ObsAuxIncrement           :      129      24
OOPS_STATS oops::ObsAuxIncrements          :       43       8
OOPS_STATS oops::ObsDataVector             :       18      12
OOPS_STATS oops::ObsDiagnostics            :        6       1
OOPS_STATS oops::ObsError                  :        3       3
OOPS_STATS oops::ObsOperator               :        6       6
OOPS_STATS oops::ObsSpace                  :        3       3
OOPS_STATS oops::ObsSpaces                 :        1       1
OOPS_STATS oops::ObsVector                 :      215     108                0.00        0.01
OOPS_STATS oops::SampledLocations          :       24       8
OOPS_STATS oops::State                     :       10       5                5.79       28.96
OOPS_STATS oops::UnstructuredInterpolator  :     6480    4320
OOPS_STATS oops::VariableChange            :        3       1
OOPS_STATS saber::ErrorCovariance          :        1       1
OOPS_STATS soca::Geometry                  :        2       2
OOPS_STATS soca::Increment                 :      139      10
OOPS_STATS soca::ModelBias                 :        4       3
OOPS_STATS soca::ModelBiasCovariance       :        1       1
OOPS_STATS soca::State                     :       16       6
OOPS_STATS ufo::BackgroundCheck            :        1       1
OOPS_STATS ufo::FinalCheck                 :        6       4
OOPS_STATS ufo::GeoVaLs                    :       90       2
OOPS_STATS ufo::ObsADT                     :        2       2
OOPS_STATS ufo::ObsADTTLAD                 :        1       1
OOPS_STATS ufo::ObsBias                    :       12       9
OOPS_STATS ufo::ObsBiasCovariance          :        3       3
OOPS_STATS ufo::ObsBoundsCheck             :        3       3
OOPS_STATS ufo::ObsComposite               :        2       2
OOPS_STATS ufo::ObsCompositeTLAD           :        1       1
OOPS_STATS ufo::ObsDomainCheck             :        8       8
OOPS_STATS ufo::ObsFilter                  :       44      21
OOPS_STATS ufo::ObsFilterData              :       18      16
OOPS_STATS ufo::ObsInsituTemperature       :        4       4
OOPS_STATS ufo::ObsInsituTemperatureTLAD   :        2       2
OOPS_STATS ufo::ObsVertInterp              :        2       2
OOPS_STATS ufo::ObsVertInterpTLAD          :        1       1
OOPS_STATS ufo::SampledLocations           :       12       7
OOPS_STATS ----------------------------- Object counts --------------------------------------
OOPS_STATS
OOPS_STATS --------------------------------------------------------------------------------------------
OOPS_STATS ------------------------------------ Timing Statistics -------------------------------------
OOPS_STATS --------------------------------------------------------------------------------------------
OOPS_STATS Name                                                :   total (ms)   count    time/call (ms)
OOPS_STATS ObsFilter::Background Check::ObsFilter              :         0.11       1            0.1114
OOPS_STATS ObsFilter::Background Check::postFilter             :         1.83       1            1.8265
OOPS_STATS ObsFilter::Background Check::preProcess             :         0.00       1            0.0006
OOPS_STATS ObsFilter::Background Check::priorFilter            :         0.00       1            0.0010
OOPS_STATS ObsFilter::Bounds Check::ObsFilter                  :         0.22       3            0.0745
OOPS_STATS ObsFilter::Bounds Check::postFilter                 :         0.00       3            0.0002
OOPS_STATS ObsFilter::Bounds Check::preProcess                 :         0.28       3            0.0931
OOPS_STATS ObsFilter::Bounds Check::priorFilter                :         0.00       3            0.0007
OOPS_STATS ObsFilter::Domain Check::ObsFilter                  :         0.73       8            0.0910
OOPS_STATS ObsFilter::Domain Check::postFilter                 :         0.00       8            0.0003
OOPS_STATS ObsFilter::Domain Check::preProcess                 :         0.78       8            0.0975
OOPS_STATS ObsFilter::Domain Check::priorFilter                :         0.44       8            0.0545
OOPS_STATS ObsFilter::Final Check::ObsFilter                   :         0.02       6            0.0040
OOPS_STATS ObsFilter::Final Check::postFilter                  :         0.03       6            0.0054
OOPS_STATS ObsFilter::Final Check::preProcess                  :         0.00       6            0.0005
OOPS_STATS ObsFilter::Final Check::priorFilter                 :         0.01       6            0.0012
OOPS_STATS ObsFilter::QCmanager::ObsFilter                     :         0.24       6            0.0406
OOPS_STATS ObsFilter::QCmanager::postFilter                    :         0.01       6            0.0024
OOPS_STATS ObsFilter::QCmanager::preProcess                    :         0.00       6            0.0002
OOPS_STATS ObsFilter::QCmanager::priorFilter                   :         0.00       6            0.0002
OOPS_STATS Vader::Vader                                        :         0.30       3            0.0997
OOPS_STATS Vader::changeVar                                    :         5.40       3            1.8009
OOPS_STATS oops::Covariance::SABER::Constructor                :        24.68       1           24.6814
OOPS_STATS oops::Covariance::SABER::multiply                   :    110034.10      12         9169.5084
OOPS_STATS oops::Diffusion::Diffusion                          :         2.58       3            0.8607
OOPS_STATS oops::Diffusion::calculateDerivedGeom               :         2.84       2            1.4198
OOPS_STATS oops::Diffusion::multiply                           :       902.34       2          451.1680
OOPS_STATS oops::Diffusion::multiplySqrtAD                     :     54574.43      36         1515.9564
OOPS_STATS oops::Diffusion::multiplySqrtTL                     :     54071.12      36         1501.9756
OOPS_STATS oops::Diffusion::setParameters                      :       159.70       3           53.2347
OOPS_STATS oops::GeoVaLs::GeoVaLs                              :         1.61      81            0.0199
OOPS_STATS oops::GeoVaLs::fill                                 :         0.44     676            0.0007
OOPS_STATS oops::GeoVaLs::fillAD                               :         0.33     468            0.0007
OOPS_STATS oops::GeoVaLs::~GeoVaLs                             :         0.28      81            0.0034
OOPS_STATS oops::Geometry::Geometry                            :      5117.20       2         2558.6008
OOPS_STATS oops::Geometry::variableSizes                       :         0.82      31            0.0265
OOPS_STATS oops::Geometry::~Geometry                           :         0.34       2            0.1718
OOPS_STATS oops::GeometryData::setGlobalTree                   :      2424.66       2         1212.3303
OOPS_STATS oops::GeometryData::setLocalTree                    :         2.62       2            1.3093
OOPS_STATS oops::GeometryData::setMeshAndTriangulation         :         1.30       2            0.6491
OOPS_STATS oops::GetValues::GetValues                          :      8657.05       9          961.8942
OOPS_STATS oops::GetValues::fillGeoVaLs                        :         0.66       9            0.0737
OOPS_STATS oops::GetValues::fillGeoVaLsAD                      :         2.97      36            0.0824
OOPS_STATS oops::GetValues::fillGeoVaLsTL                      :        84.56      36            2.3489
OOPS_STATS oops::GetValues::finalize                           :         0.35       9            0.0392
OOPS_STATS oops::GetValues::finalizeAD                         :        24.18      36            0.6716
OOPS_STATS oops::GetValues::finalizeTL                         :         0.61      36            0.0169
OOPS_STATS oops::GetValues::initialize                         :         0.20       9            0.0225
OOPS_STATS oops::GetValues::initializeAD                       :         0.38      36            0.0106
OOPS_STATS oops::GetValues::initializeTL                       :         0.96      36            0.0266
OOPS_STATS oops::GetValues::process                            :        92.82       9           10.3137
OOPS_STATS oops::GetValues::processAD                          :       188.08      36            5.2243
OOPS_STATS oops::GetValues::processTL                          :       227.09      36            6.3080
OOPS_STATS oops::Increment::Increment                          :       449.30      91            4.9373
OOPS_STATS oops::Increment::axpy                               :         5.17       1            5.1674
OOPS_STATS oops::Increment::diff                               :        17.84       3            5.9469
OOPS_STATS oops::Increment::dot_product_with                   :        35.66       3           11.8856
OOPS_STATS oops::Increment::fromFieldSet                       :        73.76      24            3.0734
OOPS_STATS oops::Increment::operator+=                         :        59.78      14            4.2703
OOPS_STATS oops::Increment::operator+=(State, Increment)       :         4.38       1            4.3779
OOPS_STATS oops::Increment::operator=                          :       231.16      50            4.6233
OOPS_STATS oops::Increment::print                              :        53.79       5           10.7574
OOPS_STATS oops::Increment::serialSize                         :         0.02      43            0.0005
OOPS_STATS oops::Increment::toFieldSet                         :         0.46      36            0.0128
OOPS_STATS oops::Increment::write                              :      5106.89       1         5106.8916
OOPS_STATS oops::Increment::zero                               :        12.11      24            0.5046
OOPS_STATS oops::Increment::~Increment                         :         4.06      91            0.0446
OOPS_STATS oops::LinearObsOper::ADT::LinearObsOperator         :         0.05       1            0.0483
OOPS_STATS oops::LinearObsOper::ADT::requiredVars              :         0.04      26            0.0014
OOPS_STATS oops::LinearObsOper::ADT::setTrajectory             :         0.64       1            0.6370
OOPS_STATS oops::LinearObsOper::ADT::simulateObsAD             :        13.46      12            1.1214
OOPS_STATS oops::LinearObsOper::ADT::simulateObsTL             :       190.64      12           15.8869
OOPS_STATS oops::LinearObsOper::ADT::~LinearObsOperator        :         0.01       1            0.0107
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::LinearObs :         0.53       1            0.5305
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::requiredV :         0.02      26            0.0006
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::setTrajec :         0.47       1            0.4668
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::simulateO :         1.15      12            0.0956
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::simulateO :         1.10      12            0.0916
OOPS_STATS oops::LinearObsOper::insitu_profile_argo::~LinearOb :         0.03       1            0.0285
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::LinearObsOpe :         0.04       1            0.0377
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::requiredVars :         0.02      26            0.0009
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::setTrajector :         0.39       1            0.3942
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::simulateObsA :         0.19      12            0.0155
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::simulateObsT :         0.15      12            0.0128
OOPS_STATS oops::LinearObsOper::temp_profile_xbt::~LinearObsOp :         0.01       1            0.0128
OOPS_STATS oops::LinearVariableChange::LinearVariableChange    :         0.24       2            0.1204
OOPS_STATS oops::LinearVariableChange::changeVarAD             :       395.62      24           16.4842
OOPS_STATS oops::LinearVariableChange::changeVarTL             :       294.41      24           12.2669
OOPS_STATS oops::LinearVariableChange::changeVarTraj           :        32.87       2           16.4360
OOPS_STATS oops::LinearVariableChange::~LinearVariableChange   :         0.14       2            0.0693
OOPS_STATS oops::LocalInterpolator::LocalInterpolator          :         8.45    6480            0.0013
OOPS_STATS oops::LocalInterpolator::apply                      :        86.73    6480            0.0134
OOPS_STATS oops::LocalInterpolator::applyAD                    :       163.55   25920            0.0063
OOPS_STATS oops::LocalInterpolator::applyTL                    :       203.08   25920            0.0078
OOPS_STATS oops::LocalInterpolator::preprocess                 :        97.06      15            6.4705
OOPS_STATS oops::LocalInterpolator::preprocessAD               :        57.98      12            4.8318
OOPS_STATS oops::LocalInterpolator::~LocalInterpolator         :         1.97    6480            0.0003
OOPS_STATS oops::ModelAuxControl::ModelAuxControl              :         0.01       4            0.0017
OOPS_STATS oops::ModelAuxControl::print                        :         0.00       3            0.0006
OOPS_STATS oops::ModelAuxControl::~ModelAuxControl             :         0.00       4            0.0006
OOPS_STATS oops::ModelAuxCovariance::ModelAuxCovariance        :         0.00       1            0.0031
OOPS_STATS oops::ModelAuxCovariance::linearize                 :         0.00       1            0.0003
OOPS_STATS oops::ModelAuxCovariance::multiply                  :         0.00      12            0.0002
OOPS_STATS oops::ModelAuxCovariance::~ModelAuxCovariance       :         0.00       1            0.0008
OOPS_STATS oops::ModelAuxIncrement::ModelAuxIncrement          :         0.06      43            0.0015
OOPS_STATS oops::ModelAuxIncrement::axpy                       :         0.00       1            0.0002
OOPS_STATS oops::ModelAuxIncrement::diff                       :         0.00       3            0.0002
OOPS_STATS oops::ModelAuxIncrement::dot_product_with           :         0.00       3            0.0003
OOPS_STATS oops::ModelAuxIncrement::operator+=                 :         0.00       2            0.0003
OOPS_STATS oops::ModelAuxIncrement::operator+=ModelAuxControl  :         0.00       1            0.0002
OOPS_STATS oops::ModelAuxIncrement::print                      :         0.01       5            0.0012
OOPS_STATS oops::ModelAuxIncrement::serialSize                 :         0.01      43            0.0001
OOPS_STATS oops::ModelAuxIncrement::write                      :         0.00       1            0.0003
OOPS_STATS oops::ModelAuxIncrement::zero                       :         0.00      12            0.0002
OOPS_STATS oops::ModelAuxIncrement::~ModelAuxIncrement         :         0.01      43            0.0003
OOPS_STATS oops::ModelData::ModelData                          :         0.00       1            0.0004
OOPS_STATS oops::ModelData::modelData                          :         0.00       1            0.0004
OOPS_STATS oops::ModelData::~ModelData                         :         0.00       1            0.0009
OOPS_STATS oops::ObsAuxControl::ObsAuxControl                  :         1.89      12            0.1573
OOPS_STATS oops::ObsAuxControl::print                          :         0.61       9            0.0673
OOPS_STATS oops::ObsAuxControl::requiredHdiagnostics           :         0.00       6            0.0003
OOPS_STATS oops::ObsAuxControl::requiredVars                   :         0.01      18            0.0004
OOPS_STATS oops::ObsAuxControl::write                          :         0.05       3            0.0154
OOPS_STATS oops::ObsAuxControl::~ObsAuxControl                 :         0.02      12            0.0013
OOPS_STATS oops::ObsAuxCovariance::ObsAuxCovariance            :         0.21       3            0.0715
OOPS_STATS oops::ObsAuxCovariance::linearize                   :         0.02       3            0.0057
OOPS_STATS oops::ObsAuxCovariance::multiply                    :         0.13      36            0.0037
OOPS_STATS oops::ObsAuxCovariance::write                       :         0.06       3            0.0215
OOPS_STATS oops::ObsAuxCovariance::~ObsAuxCovariance           :         0.01       3            0.0028
OOPS_STATS oops::ObsAuxIncrement::ObsAuxIncrement              :         2.57     129            0.0199
OOPS_STATS oops::ObsAuxIncrement::axpy                         :         0.01       3            0.0037
OOPS_STATS oops::ObsAuxIncrement::diff                         :         0.02       9            0.0025
OOPS_STATS oops::ObsAuxIncrement::dot_product_with             :         0.01       9            0.0011
OOPS_STATS oops::ObsAuxIncrement::operator+=                   :         0.02       6            0.0037
OOPS_STATS oops::ObsAuxIncrement::print                        :         0.01      15            0.0005
OOPS_STATS oops::ObsAuxIncrement::serialSize                   :         0.02     258            0.0001
OOPS_STATS oops::ObsAuxIncrement::zero                         :         0.02      36            0.0005
OOPS_STATS oops::ObsAuxIncrement::~ObsAuxIncrement             :         0.07     129            0.0005
OOPS_STATS oops::ObsDataVector::ObsDataVector                  :         0.12      18            0.0066
OOPS_STATS oops::ObsDataVector::mask                           :         0.01       6            0.0009
OOPS_STATS oops::ObsDataVector::operator=                      :         0.02       9            0.0018
OOPS_STATS oops::ObsDataVector::print                          :         5.78       6            0.9635
OOPS_STATS oops::ObsDataVector::save                           :         1.63      12            0.1354
OOPS_STATS oops::ObsDataVector::~ObsDataVector                 :         0.02      18            0.0013
OOPS_STATS oops::ObsDiagnostics::ObsDiagnostics                :         0.10       6            0.0172
OOPS_STATS oops::ObsDiagnostics::~ObsDiagnostics               :         0.03       6            0.0057
OOPS_STATS oops::ObsError::getRMSE                             :         0.56       8            0.0706
OOPS_STATS oops::ObsError::inverseMultiply                     :         0.03      36            0.0009
OOPS_STATS oops::ObsError::obserrors                           :         1.81      14            0.1289
OOPS_STATS oops::ObsError::update                              :        46.37       6            7.7284
OOPS_STATS oops::ObsError::~ObsError                           :         0.11       3            0.0353
OOPS_STATS oops::ObsOperator::ADT::ObsOperator                 :         2.29       2            1.1445
OOPS_STATS oops::ObsOperator::ADT::computeReducedVars          :         0.17       3            0.0581
OOPS_STATS oops::ObsOperator::ADT::locations                   :         0.57       3            0.1901
OOPS_STATS oops::ObsOperator::ADT::requiredVars                :         0.00       3            0.0010
OOPS_STATS oops::ObsOperator::ADT::simulateObs                 :        62.39       2           31.1934
OOPS_STATS oops::ObsOperator::ADT::~ObsOperator                :         0.65       2            0.3231
OOPS_STATS oops::ObsOperator::insitu_profile_argo::ObsOperator :         5.20       2            2.6020
OOPS_STATS oops::ObsOperator::insitu_profile_argo::computeRedu :         0.04       3            0.0129
OOPS_STATS oops::ObsOperator::insitu_profile_argo::locations   :         0.44       3            0.1455
OOPS_STATS oops::ObsOperator::insitu_profile_argo::requiredVar :         0.00       3            0.0006
OOPS_STATS oops::ObsOperator::insitu_profile_argo::simulateObs :        23.59       2           11.7938
OOPS_STATS oops::ObsOperator::insitu_profile_argo::~ObsOperato :         0.03       2            0.0158
OOPS_STATS oops::ObsOperator::temp_profile_xbt::ObsOperator    :         1.38       2            0.6880
OOPS_STATS oops::ObsOperator::temp_profile_xbt::computeReduced :         0.04       3            0.0128
OOPS_STATS oops::ObsOperator::temp_profile_xbt::locations      :         0.40       3            0.1350
OOPS_STATS oops::ObsOperator::temp_profile_xbt::requiredVars   :         0.00       3            0.0010
OOPS_STATS oops::ObsOperator::temp_profile_xbt::simulateObs    :         0.05       2            0.0229
OOPS_STATS oops::ObsOperator::temp_profile_xbt::~ObsOperator   :         0.02       2            0.0084
OOPS_STATS oops::ObsSpace::ObsSpace                            :      1124.98       3          374.9942
OOPS_STATS oops::ObsSpace::obsvariables                        :         0.00      12            0.0001
OOPS_STATS oops::ObsSpace::save                                :       978.77       3          326.2559
OOPS_STATS oops::ObsSpace::~ObsSpace                           :         0.18       3            0.0613
OOPS_STATS oops::ObsVector::ObsVector                          :         0.41     209            0.0020
OOPS_STATS oops::ObsVector::axpy                               :         0.08     228            0.0004
OOPS_STATS oops::ObsVector::dot_product                        :        40.12     242            0.1658
OOPS_STATS oops::ObsVector::info                               :         2.34      18            0.1301
OOPS_STATS oops::ObsVector::operator*=                         :         0.04     132            0.0003
OOPS_STATS oops::ObsVector::operator+=                         :         0.02      90            0.0003
OOPS_STATS oops::ObsVector::operator-=                         :         0.00       6            0.0006
OOPS_STATS oops::ObsVector::operator=                          :         0.09     144            0.0006
OOPS_STATS oops::ObsVector::rms                                :         0.27       8            0.0337
OOPS_STATS oops::ObsVector::save                               :         0.90      18            0.0500
OOPS_STATS oops::ObsVector::zero                               :         0.02      45            0.0003
OOPS_STATS oops::ObsVector::~ObsVector                         :         0.15     209            0.0007
OOPS_STATS oops::Parameters::deserialize                       :         9.13     173            0.0528
OOPS_STATS oops::Parameters::serialize                         :         1.83      67            0.0273
OOPS_STATS oops::SampledLocations::SampledLocations            :         0.00      12            0.0003
OOPS_STATS oops::SampledLocations::latitudes                   :         0.00       9            0.0002
OOPS_STATS oops::SampledLocations::longitudes                  :         0.00       9            0.0001
OOPS_STATS oops::SampledLocations::times                       :         0.00       9            0.0001
OOPS_STATS oops::SampledLocations::~SampledLocations           :         0.41      24            0.0170
OOPS_STATS oops::State::State                                  :      9850.83      10          985.0826
OOPS_STATS oops::State::print                                  :        22.34       3            7.4480
OOPS_STATS oops::State::toFieldSet                             :         0.10       5            0.0193
OOPS_STATS oops::State::write                                  :      3934.34       1         3934.3364
OOPS_STATS oops::State::~State                                 :         0.40      10            0.0403
OOPS_STATS oops::UnstructuredInterpolator::UnstructuredInterpo :         3.66    6480            0.0006
OOPS_STATS oops::UnstructuredInterpolator::apply               :       271.13   32400            0.0084
OOPS_STATS oops::UnstructuredInterpolator::applyAD             :       148.08   25920            0.0057
OOPS_STATS oops::VariableChange::VariableChange                :         0.73       3            0.2423
OOPS_STATS oops::VariableChange::changeVar                     :        58.98       3           19.6611
OOPS_STATS oops::VariableChange::~VariableChange               :         0.03       3            0.0112
OOPS_STATS oops::util::FieldSetHelpers::readFieldSet           :      5998.05       3         1999.3502
OOPS_STATS saber::ErrorCovariance::ErrorCovariance             :      8712.67       1         8712.6673
OOPS_STATS saber::ErrorCovariance::doMultiply                  :    109342.51      12         9111.8761
OOPS_STATS saber::ErrorCovariance::~ErrorCovariance            :         0.00       1            0.0002
OOPS_STATS soca::Balance::Balance                              :        14.64       1           14.6395
OOPS_STATS soca::Balance::multiply                             :       113.23      12            9.4361
OOPS_STATS soca::Balance::multiplyAD                           :       103.17      12            8.5972
OOPS_STATS soca::BkgErrFilt::BkgErrFilt                        :         0.24       1            0.2405
OOPS_STATS soca::BkgErrFilt::multiply                          :        63.48      24            2.6451
OOPS_STATS soca::FieldsMetadata::FieldsMetadata                :        26.82       2           13.4122
OOPS_STATS soca::LinearModel2GeoVaLs::multiply                 :        19.36      12            1.6132
OOPS_STATS soca::LinearModel2GeoVaLs::multiplyAD               :        28.10      12            2.3417
OOPS_STATS soca::Model2GeoVaLs::Model2GeoVaLs                  :         0.00       3            0.0002
OOPS_STATS soca::Model2GeoVaLs::changeVar                      :         7.96       3            2.6538
OOPS_STATS soca::ParametricOceanStdDev::ParametricOceanStdDev  :      2674.58       1         2674.5755
OOPS_STATS soca::ParametricOceanStdDev::multiply               :        34.84      12            2.9032
OOPS_STATS soca::ParametricOceanStdDev::multiplyAD             :        26.84      12            2.2369
OOPS_STATS soca::VariableChange::VariableChange                :         0.70       3            0.2342
OOPS_STATS soca::VariableChange::changeVar                     :        58.93       3           19.6443
OOPS_STATS soca::readNcAndInterp::readNcAndInterp              :      1599.43       1         1599.4294
OOPS_STATS ufo::ObsFilter::ObsFilter                           :         1.36      24            0.0568
OOPS_STATS ufo::ObsFilter::checkFilterData                     :         0.02      72            0.0003
OOPS_STATS ufo::ObsFilter::postFilter                          :         1.90      24            0.0791
OOPS_STATS ufo::ObsFilter::preProcess                          :         1.09      24            0.0454
OOPS_STATS ufo::ObsFilter::priorFilter                         :         0.48      24            0.0199
OOPS_STATS ufo::ObsFilter::~ObsFilter                          :         6.86      44            0.1560
OOPS_STATS util::Timers::Total                                 :    158434.32       1       158434.3198
OOPS_STATS util::Timers::measured                              :    158091.14       1       158091.1382
OOPS_STATS ------------------------------------ Timing Statistics -------------------------------------

OOPS_STATS ------------------------------------------------------------------------------------------------------------------
OOPS_STATS ---------------------------------- Parallel Timing Statistics ( 720 MPI tasks) -----------------------------------
OOPS_STATS ------------------------------------------------------------------------------------------------------------------
OOPS_STATS Name                                                :     min (ms)    max (ms)    avg (ms)     % total   imbal (%)
OOPS_STATS oops::Covariance::SABER::multiply                   :    109767.03   110156.43   110029.77       69.45        0.35
OOPS_STATS oops::Diffusion::multiply                           :       884.45     1108.15      922.03        0.58       24.26
OOPS_STATS oops::Diffusion::multiplySqrtAD                     :     54263.73    54629.87    54437.90       34.36        0.67
OOPS_STATS oops::Diffusion::multiplySqrtTL                     :     54036.87    54247.11    54157.25       34.18        0.39
OOPS_STATS oops::Geometry::Geometry                            :      4872.53     5199.99     5154.04        3.25        6.35
OOPS_STATS oops::GeometryData::setGlobalTree                   :      2078.45     2539.82     2437.84        1.54       18.93
OOPS_STATS oops::GetValues::GetValues                          :        93.44     9924.36     4576.87        2.89      214.80
OOPS_STATS oops::GetValues::fillGeoVaLs                        :         0.42     1533.88      629.05        0.40      243.77
OOPS_STATS oops::GetValues::processAD                          :       187.15      218.53      191.55        0.12       16.39
OOPS_STATS oops::GetValues::processTL                          :       227.09      273.42      234.19        0.15       19.78
OOPS_STATS oops::Increment::Increment                          :       106.05      510.83      445.02        0.28       90.96
OOPS_STATS oops::Increment::operator=                          :       107.09      308.43      222.54        0.14       90.47
OOPS_STATS oops::Increment::write                              :       696.64     5106.89     2973.53        1.88      148.32
OOPS_STATS oops::LinearObsOper::ADT::simulateObsTL             :       126.50      245.72      184.58        0.12       64.59
OOPS_STATS oops::LinearVariableChange::changeVarAD             :       131.57      451.70      383.59        0.24       83.45
OOPS_STATS oops::LinearVariableChange::changeVarTL             :       117.91      359.47      315.13        0.20       76.65
OOPS_STATS oops::LocalInterpolator::applyAD                    :       162.18      193.68      166.64        0.11       18.90
OOPS_STATS oops::LocalInterpolator::applyTL                    :       203.08      248.88      209.35        0.13       21.88
OOPS_STATS oops::LocalInterpolator::preprocess                 :        50.32     3294.90     1154.98        0.73      280.92
OOPS_STATS oops::ObsOperator::ADT::simulateObs                 :        31.05      576.08      516.39        0.33      105.55
OOPS_STATS oops::ObsSpace::ObsSpace                            :      1013.05     1275.21     1095.54        0.69       23.93
OOPS_STATS oops::ObsSpace::save                                :       870.37     1089.26      981.91        0.62       22.29
OOPS_STATS oops::State::State                                  :      8527.38    18334.06    13952.55        8.81       70.29
OOPS_STATS oops::State::print                                  :        22.34     4434.07     2156.68        1.36      204.56
OOPS_STATS oops::State::write                                  :       461.82     3934.34     1803.05        1.14      192.59
OOPS_STATS oops::UnstructuredInterpolator::apply               :       271.13      334.16      278.60        0.18       22.62
OOPS_STATS oops::util::FieldSetHelpers::readFieldSet           :      5771.66     5998.05     5964.72        3.76        3.80
OOPS_STATS saber::ErrorCovariance::ErrorCovariance             :      8547.68     8719.85     8648.01        5.46        1.99
OOPS_STATS saber::ErrorCovariance::doMultiply                  :    109203.46   109597.16   109332.77       69.01        0.36
OOPS_STATS util::Timers::Total                                 :    158423.54   158434.32   158430.66      100.00        0.01
OOPS_STATS util::Timers::measured                              :    157893.36   158231.41   158163.24       99.83        0.21
OOPS_STATS ---------------------------------- Parallel Timing Statistics ( 720 MPI tasks) -----------------------------------

OOPS_STATS Run end                                  - Runtime:    166.68 sec,  Memory: total:   673.33 Gb, per task: min =   870.04 Mb, max =  4374.32 Mb
Run: Finishing oops::Variational<SOCA, UFO and IODA observations> with status = 0
OOPS Ending   2025-12-30 14:50:56 (UTC-0500)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use vader for temperature linear variable change

5 participants